Memory buffers for merging local data from memory modules

ABSTRACT

An integrated circuit to serialize local data and selectively merge it with serialized feed-through data into a serial data stream output that includes a parallel-in-serial-out (PISO) shift register, a multiplexer, and a transmitter. The PISO shift register serializes parallel data on a local data bus into serialized local data. The multiplexer selectively merges serialized local data and feed-through data into a serial data stream. The transmitter drives the serial data stream onto a serial data link. In another embodiment of the invention, a method for a memory module includes receiving an input serial data stream; merging local frames of data and feed-through frames of data together into an output serial data stream in response to a merge enable signal; and transmitting the output serial data stream on a northbound data output to a next memory module or a memory controller. Other embodiments of the invention are disclosed and claimed.

FIELD

Embodiments of the invention relate generally to memory, andspecifically to merging data from a memory buffer onto serial datachannels.

BACKGROUND INFORMATION

In memory circuits there is typically a memory read latency that is thetime period it takes for valid data to be read out of a memory circuit.A memory write latency is typically also required that is the timeperiod to hold valid data for a memory circuit to write the data intomemory. The memory read latency and the memory write latency maysometimes be buffered from a processor by a cache memory. However, thereare occasions when the desired data is not found in the cache memory. Inthose cases, a processor may need to then read or write data with thememory circuits. Thus, the respective memory read latency or memorywrite latency may be experienced by the processor. If memory circuitsdiffer, the memory read latencies and memory write latency may beinconsistent from one memory circuit to the next. In which case, thememory read latency and memory write latency experienced by a processorwill differ.

Previously, memory modules were plugged into a mother or host printedcircuit board and coupled in parallel to a parallel data bus over whichparallel data could be read from and written into memory. The paralleldata bus had parallel data bit lines that were synchronized together totransfer one or more data bytes or words of data at a time. The paralleldata bit lines are typically routed over a distance on a printed circuitboard (PCB) from one memory module socket to another. This introduces afirst parasitic capacitive load. As the memory modules are plugged intoa memory socket, an additional parasitic capacitive load is introducedonto the parallel data bits lines of the parallel data bus. As there maybe a number of memory modules plugged in, the additional parasiticcapacitive load may be significant and bog down high frequency memorycircuits.

One memory module is typically addressed by an address on address linesat a time. The one addressed memory module, typically writes data ontothe parallel data bus at a time. Other memory modules typically have towait to write data onto the parallel data bus in order to avoidcollisions.

While parallel data bit lines may speed data flow in certain instances,a parallel data bus in a memory may slow the read and write access ofdata between a memory circuit and a processor.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a block diagram of a typical computer system inwhich embodiments of the invention may be utilized.

FIG. 1B illustrates a block diagram of a client-server system in whichembodiments of the invention may be utilized.

FIG. 2A illustrates a block diagram of a central processing unit inwhich embodiments of the invention may be utilized.

FIG. 2B illustrates a block diagram of another central processing unitin which embodiments of the invention may be utilized.

FIG. 3 illustrates a simplified block diagram of a buffered memorycontroller to couple data into and out of banks of buffered memorymodules.

FIG. 4 illustrates a block diagram of a buffered memory module includinga buffer that may merge data with feed through data.

FIG. 5 illustrates a detailed block diagram of a buffered memorycontroller coupling to a bank of buffered memory modules.

FIG. 6 (FIGS. 6-1 and 6-2) illustrates a functional block diagram of abuffer of a buffered memory module.

FIG. 7A illustrates a simplified block diagram of the data merge logicincluding lanes of data merge logic slices coupled to transmitters.

FIG. 7B illustrates a schematic diagram of a data merge logic slice forone lane of serial data.

FIG. 8 illustrates a timing diagram of signals for a data merge logicslice functioning in a twelve bit mode.

FIG. 9 illustrates a timing diagram of signals for a data merge logicslice functioning in a six-bit mode.

FIG. 10 illustrates a flow chart for the initialization, training, andfunctioning of the buffer in merging local data and feed through datatogether into a serial data stream output.

DETAILED DESCRIPTION

In the following detailed description of embodiments of the invention,numerous specific details are set forth in order to provide a thoroughunderstanding of the invention. However, it will be obvious to oneskilled in the art that the embodiments of the invention may bepracticed without these specific details. In other instances well knownmethods, procedures, components, and circuits have not been described indetail so as not to unnecessarily obscure aspects of the embodiments ofthe invention.

Generally the embodiments of the invention provide a data merge feature,referred to as a Northbound Data Merge (NBDM), that replaces parts ofthe data on a high speed link with its own data, on the fly. That is,the embodiments of the invention replace part of the incoming serialdata traffic (e.g., “idle packets or frames”) over a serial data linkwith its local data, without having internal core logic process (e.g.,serial-to-parallel conversion, assemblage into frames, anddepacketize/deinterleave data) the incoming serial data traffic todetermine where to insert the local data and retransmit the incomingdata traffic with the local data inserted therein.

Previously, the incoming serial data had to be assembled into frames andreceived by the core logic in order to transmit local data. Withouthaving to process the incoming serial data in order to transmit localdata, an input/output (IO) interface of a memory module may simplyretransmit the incoming serial data stream received from other memorymodules or the memory controller over the serial data link, bypassingthe internal core logic of a buffer integrated circuit. This can reducedata latency in the serial data stream. The portion of the serial datastream that is to be retransmitted is sometimes referred to as“Feed-thru Data” or “Feed-through data” (FTD).

Without any local data to transmit, the IO interface normallyretransmits the received serial data stream, bypassing the core logic ofthe chip. When the core logic of the buffer memory chip needs totransmit local data, it sends a merge request along with the local datato the IO interface. Because the core clock that generates the localdata is aligned during training to a frame clock of the high speedserial data link in the embodiments of the invention, the IO interfacecan readily merge the data at the appropriate frame boundary to replacean idle packet or frame.

Previously it was contemplated that the received serial data would beassembled into frames and received by the core logic and thenre-transmitted on the outbound link. In this case, if the core logic hadlocal data to send on the outbound link, it would then replace some ofthe incoming data with its own data, repacketize and serialize the dataonto the outbound link. This would incur a data latency of at least twoframes of data. The embodiments of the invention set up the merge timingduring initial training so that the local data can be merged into theoutbound link without having to receive and analyze the incoming dataduring normal operation in order to replace idle packets. Theembodiments of the invention can reduce the data latency through thebuffer memory integrated circuit from at least two frames of data downto a few bit intervals.

In one embodiment of the invention, an integrated circuit is providedthat includes a serial input/output interface with one or more lanes.Each lane of the serial communication channel may include a firstparallel-in-serial-output (PISO) shift register, a first multiplexer,and a serial transmitter coupled together.

The first parallel-in-serial-output (PISO) shift register has a parallelinput coupled to a local data bus, a clock input coupled to a firstclock signal, a load/shift-bar input coupled to a load signal. The firstPISO shift register serializes parallel data on the local data bus intoserialized local data on a first serial output.

The first multiplexer has a first data input coupled to the first serialoutput, a second data input to receive feed-through data, and a firstselect control input coupled to a local data select signal. Themultiplexer selectively merges the serialized local data and thefeed-through data into a serial data stream on a multiplexed output inresponse to the local data select signal.

The serial transmitter has an input coupled to the multiplexed output ofthe multiplexer to receive the serial data stream. The serialtransmitter drives the serial data stream onto a serial data link.

The feed-through data may be two bits wide while the parallel input tothe PISO shift register may be six bits wide and the serial output ofthe PISO shift register may be two bits wide. In this case, the firstmultiplexer may be a two bit bus multiplexer such that the serial datastream at the multiplexed output is two bits wide so that the serialtransmitter receives a two bit serial data stream and serializes it ontothe serial data link as a single bit serial data stream.

Each lane may further include a second multiplexer having a first inputto receive resynchronized data, a second input to receive re-sampleddata, and a select input coupled to a local clock mode signal. Thesecond multiplexer to select between output the re-sampled data or theresynchronized data as the feed-through data in response to the localclock mode signal. Each lane may further include control logic coupledto the first multiplexer and the first PISO shift register. The controllogic may include merge control logic and mode control logic. Thecontrol logic may receive the first clock signal and a merge enablesignal to generate the local data select signal to merge the serializedlocal data and the feed-through data into the serial data stream inresponse to the merge enable signal and the first clock signal.

In another embodiment of the invention, a method for a memory module isprovided including receiving an input serial data stream representingfeed-through frames of data interspersed between idle frames of data;merging local frames of data and the feed-through frames of datatogether into an output serial data stream in response to a merge enablesignal without decoding the input serial data stream; and transmittingthe output serial data stream on a northbound data output to a nextmemory module or a memory controller. The local frames of data can bemerged into the output serial data stream by replacing idle frames ofdata in the input serial data stream. In receiving the input serial datastream, sampling (also may be referred to as re-sampling) of the bits ofdata in the input serial data stream or re-synchronizing the bits ofdata in the input serial data stream may be provided. In merging of thelocal frames of data and the feed-through frames of data together,serializing parallel bits of the local frames of data into serial bitsof data and multiplexing the serial bits of data of the local frames ofdata and serial bits of the feed-through frames of data into serial bitsof the output serial data stream in response to the merge enable signalmay be provided. A local frame of data may be selectively received inparallel over a local bus in six bit or twelve bit packets in responseto a bus mode signal.

In another embodiment of the invention, a system is provided including:a processor, a memory controller coupled to the processor and at leastone bank of memory coupled to the memory controller. The processor isprovided to execute instruction and process data. The memory controlleris provided to receive write memory instructions with write data fromthe processor and to receive read memory instructions from the processorand supply read data thereto.

The one bank of memory includes one or more memory modules each of whichhas a buffer integrated circuit and a random access memory integratedcircuit coupled together. The buffer integrated circuit includes asouthbound serial input/output interface with one or more serial lanesto receive the write data from the memory controller, and a northboundserial input/output interface with one or more serial lanes of anorthbound serial input and a northbound serial output to transmit theread data to the memory controller.

Each serial lane in the northbound input/output interface has aparallel-to-serial converter and a first multiplexer. Theparallel-to-serial converter has a parallel input coupled to parallelbits of a local data bus, a clock input coupled to a first clock signal,a load/shift-bar input coupled to a load signal. The parallel-to-serialconverter serializes the parallel bits of data on the local data businto serialized local data on a first serial output. The firstmultiplexer has a first data input coupled to the serial output of theparallel-to-serial converter, a second data input to receive serialfeed-through data from the northbound serial input, and a select inputcoupled to a local data select signal. The first multiplexer toselectively merge the serialized local data and the serial feed-throughdata into a serial data stream on the northbound serial output inresponse to the local data select signal.

Each serial lane in the northbound serial input/output interface mayfurther have a transmitter to drive the serial data stream onto thenorthbound serial data output towards the memory controller with aninput coupled to the multiplexed output of the first multiplexer toreceive the serial data stream, the transmitter.

Each serial lane of the northbound serial input/output interface mayfurther include control logic coupled to the multiplexer and the firstparallel-to-serial converter. The control logic to receive the firstclock signal and a merge enable signal in order to generate the localdata select signal and merge the serialized local data and the serialfeed-through data into the serial data stream in response to the mergeenable signal and the first clock signal.

For each bank of memory in the system, the memory controller includes anorthbound serial input interface to receive one or more lanes of serialdata from the one or more memory modules, and a southbound serial outputinterface to transmit one or more lanes of serial data to the one ormore memory modules.

In another embodiment of the invention, a buffered memory module isprovided including a printed circuit board, a plurality of random accessmemory (RAM) integrated circuits, and a buffer integrated circuit. Theprinted circuit board has an edge connection to couple to a receptacleof a host system. The plurality of random access memory (RAM) integratedcircuits and the buffer integrated circuit are coupled to the printedcircuit board. The buffer integrated circuit is electrically coupled tothe plurality of RAM integrated circuits and the edge connection. Thebuffer integrated circuit has a southbound input/output interface and anorthbound input/output interface with data merge logic having aplurality of merge logic slices for a plurality of lanes of serial datastreams.

Each merge logic slice of the buffer integrated circuit includes a firstparallel-in-serial-output (PISO) shift register and a first multiplexer.The first parallel-in-serial-output (PISO) shift register has a parallelinput coupled to a local data bus, a clock input coupled to a firstclock signal, a load/shift-bar input coupled to a first load signal. Thefirst PISO shift register to serialize parallel data on the local databus into serialized local data on a first serial output. The firstmultiplexer has a first data input coupled to the first serial output ofthe first PISO shift register, a second data input to receive serializedfeed-through data, and a first select input coupled to a local dataselect signal. The first multiplexer selectively merges the serializedlocal data and the serialized feed-through data into a serial datastream on a multiplexed output in response to the local data selectsignal.

Each merge logic slice may further include control logic coupled to thefirst multiplexer and the first PISO shift register. The control logicreceives the first clock signal and a merge enable signal to generatethe local data select signal to merge the serialized local data and theserialized feed-through data into the serial data stream in response tothe merge enable signal and the first clock signal.

The northbound input/output interface of the buffer integrated circuitin the buffered memory module may further include a plurality oftransmitters each having an input coupled to a corresponding output ofthe first multiplexer in each merge logic slice, the plurality oftransmitters to receive the serial data stream and drive it onto aserial data link.

In another embodiment of the invention, a memory system is providedincluding a plurality of buffered memory modules daisy chained togetherto form a bank of memory. Each buffered memory module includes aplurality of memory integrated circuits, and a buffer integrated circuitcoupled to the plurality of memory integrated circuits. The bufferintegrated circuit includes a southbound input/output serial interfaceto receive and retransmit southbound serial data from a memorycontroller or a prior buffered memory module to a next buffered memorymodule, a northbound input/output serial interface to receive northboundserial data from at least one buffered memory module as serializedfeed-through data and retransmit it out towards the memory controller, awrite data first-in-first-out (FIFO) buffer to store write data from thesouthbound input/output serial interface addressed to the bufferedmemory module by a write command, memory input/output interface totransfer write data stored in the write data FIFO buffer into at leastone of the plurality of memory integrated circuits and to transfer readdata from at least one of the plurality of memory integrated circuitsinto a read data FIFO buffer, and the read data FIFO buffer to storeread data from at least one of the plurality of memory integratedcircuits as the local data addressed from the buffered memory module bya read command.

The northbound input/output serial interface serializes the local datafrom the plurality of memory integrate circuits and merges it into anorthbound serial data stream with the serialized feed-through data on atiming basis without decoding the received northbound serial data. Thenorthbound input/output serial interface of includes a third FIFObuffer, data merge logic coupled to the third FIFO buffer, and aplurality of transmitters coupled to the data merge logic.

The data merge logic has a plurality of merge logic slices eachincluding a first parallel-in-serial-output (PISO) shift register toserialize parallel data on the local data bus into serialized local dataon a first serial output, and a first multiplexer to selectively mergeserialized local data and serialized feed-through data into a serialdata stream on a multiplexed output in response to the local data selectsignal. The PISO shift register has a parallel input coupled to a localdata bus, a clock input coupled to a first clock signal, and aload/shift-bar input coupled to a first load signal. The firstmultiplexer having a first data input coupled to the first serial outputof the first PISO shift register, a second data input to receiveserialized feed-through data, a first select input coupled to a localdata select signal, the first multiplexer.

Each of the plurality of transmitters has an input coupled to acorresponding output of the first multiplexer in each merge logic slice.The plurality of transmitters receive data from the serial data streamand drive it onto a serial data link.

In the memory system, each merge logic slice of the data merge logic mayfurther include control logic coupled to the first multiplexer and thefirst PISO shift register to receive the first clock signal and a mergeenable signal and generate the local data select signal to merge theserialized local data and the serialized feed-through data into theserial data stream.

The memory system may further include a memory controller coupled to atleast one of the plurality of buffered memory modules. The memorycontroller has a southbound output serial interface to transmit thesouthbound serial data stream to the at least one of the plurality ofbuffered memory modules and a northbound input serial interface toreceive the northbound serial data stream from the least one of theplurality of buffered memory modules.

Referring now to FIG. 1A, a block diagram of a typical computer system100 in which embodiments of the invention may be utilized isillustrated. The computer system 100A includes a central processing unit(CPU) 101; input/output devices (I/O) 102 such as keyboard, modem,printer, external storage devices and the like; and monitoring devices(M) 103, such as a CRT or graphics display. The monitoring devices (M)103 may provide computer information in a human intelligible format suchas visual or audio formats. The system 100 may be a number of differentelectronic systems other than a computer system.

Referring now to FIG. 1B, a client server system 100B in whichembodiments of the invention may be utilized is illustrated. The clientserver system 100B includes one or more clients 110A-110M coupled to anetwork 112 and a server 114 coupled to the network 112. The clients110A-110M communicate with the server 114 through the network 112 inorder to transmit or receive information and gain access to any databaseand/or application software that may be needed on the server. The server114 has a central processing unit with memory and may further includeone or more disk drive storage devices. The server 114 may be used in astorage area network (SAN) as a network attached storage (NAS) device,for example, and have an array of disks. The data access to the server114 is shared over the network 112 with the multiple clients 110A-110C.

Referring now to FIG. 2A, a block diagram of a central processing unit101A in which embodiments of the invention may be utilized isillustrated. The central processing unit 101A includes a processor 201,a memory controller 202, and a first memory 204A of a first memorychannel coupled together as shown and illustrated. The centralprocessing unit 101A may further include a cache memory 203, coupledbetween the memory controller 202 and the processor 201 and a diskstorage device 206 coupled to the processor 201. The central processingunit 101A may further include a second memory channel with a secondmemory 204B coupled to the memory controller 202. As illustrated by thecentral processing unit 101A, the memory controller 202 and the cachememory 203 may be external to the processor 201.

Referring now to FIG. 2B, a block diagram of another central processingunit 101B in which embodiments of the invention may be utilized isillustrated. The central processing unit 101B includes a processor 201′with an internal memory controller 202′ and a first memory channel witha memory 204A coupled to the internal memory controller 202′ of theprocessor 201′. The processor 201′ may further include an internal cachememory 203′. The central processing unit 101B may further include asecond memory 204B for a second memory channel, and a disk storagedevice 206 coupled to the processor 201′.

The disk storage device 206 may be a floppy disk, zip disk, DVD disk,hard disk, rewritable optical disk, flash memory or other non-volatilestorage device.

The processor 201,201′ may further include one or more execution unitsand one or more levels of cache memory. Other levels of cache memory maybe external to the processor and interface to the memory controller. Theprocessor, the one or more execution units, or the one or more levels ofcache memory may read or write data (including instructions) through thememory controller with the memory 204A-204B. In interfacing to thememory controller 202,202′, there may be address, data, control andclocking signals coupled to the memory as part of the memory interface.The processors 201,201′ and the disk storage device 206 may both readand write information into the memories 204A,204B.

Each of the memories 204A and 204B illustrated in FIGS. 2A-2B mayinclude one or more buffered memory modules (MM1-MMn), such as a fullybuffered (FB) dual in-line memory module (DIMM), (FBDIMM), or a fullybuffered (FB) single in-line memory module (SIMM), (FBSIMM), forexample.

The memory controller 202,202′ interfaces to each memory 204A-240B. Inone embodiment of the invention, the memory controller 202,202′particularly interfaces to a buffer (not shown in FIGS. 2A-2B, but seebuffer 450A in FIG. 5) in a first buffered memory module MM1 of eachmemory 204A-204B. With the memory controller 202,202′ interfacing to thebuffers of the memory modules, direct interface to the memory devices ofthe buffered memory modules (MM1-MMn) can be avoided. In this manner,different types of memory devices may be used to provide memory storagewhile the interface between the buffer and the memory controller canremain consistent.

Referring now to FIG. 3, a buffered memory module (BMM) memorycontroller (BMMMC) 302 coupled to one or more memory banks 304A-304F(generally referred to as memory bank 304 or memory banks 304) isillustrated. The memory controller 302 can support more than twochannels of memory and more than two memory banks of memory modules.Each memory bank 304 is made up of a plurality of buffered memorymodules 310A-310H coupled together in a serial chain. This serial chainof buffered memory modules 310A-310H is also sometimes referred to as adaisy chain. Adjacent memory modules are coupled to each other,sometimes referred to as being daisy-chained together, such as memorymodule 310A being coupled to adjacent memory module 310B, for example.

Each of the memory modules 310A-310H in each bank bidirectionallycommunicate in a serial fashion with the memory controller 302 along theserial chain of memory modules 310A-310H. There is a southbound serialdata link (SB) from the memory controller 302 to each memory bank 304that may also be referred to as an outbound data link with outboundcommands (e.g., read and write) and data. All write data from the memorycontroller that is to be written into the memory modules is sent overthe southbound serial data link. There is a northbound serial data link(NB) from each memory bank 304 to the memory controller 302 that mayalso be referred to as an inbound data link with inbound data. All readdata from the memory modules is sent to the memory controller over thenorthbound serial data link.

In the southbound serial data link (SB), data out from the memorycontroller 302 to a memory bank 304 is first coupled to the first memorymodule 310A which can read the data and pass it to memory module 310B.Memory module 310B can read the data and pass it to the next memorymodule in the serial chain, and so on and so forth until the last memorymodule in the southbound serial chain is reached. The last memory modulein the southbound serial chain, memory module 310H, has no furthermemory module to pass data to and so the southbound serial data linkterminates.

In the northbound serial data link (NB), data is serially communicatedin a direction from the memory bank 304 to the memory controller 302.Each memory module in each memory bank communicates back towards thememory controller on the northbound serial data link (NB). Memory module310H begins a serial chain of memory modules passing data towards thememory controller. Serial data transmitted by memory module 310H passesthrough or is otherwise retransmitted by memory module 310G. Whilememory module 310G may pass or retransmit the serial data from the priormemory module 310H, it may also add or merge its own local data to thenorthbound serial data stream heading to the memory controller 302.Similarly, each memory module down the chain passes or retransmits theserial data from the prior memory module and may add or merge their ownlocal data to the northbound serial data stream heading to the memorycontroller 302. The last memory module in the northbound serial chain,memory module 310A, transmits the final northbound serial data stream tothe memory controller 302.

The northbound and southbound serial data links may be considered asproviding point to point communication from one memory module to anothermemory module and so on and so forth along the serial chain. The serialdata flow from the memory controller 302 out to memory module 310Athrough memory module 310H may be referred to as a south data flow. Theserial data flow from memory module 310H through memory module 310Z tothe memory controller 302 may be referred to as a northbound data flow.In FIG. 3, the southbound data flow is illustrated by an arrow labeledSB, while the northbound data flow is illustrated by an arrow labeledNB.

Referring now to FIG. 4, a buffered memory module (BMM) 310 isillustrated that is exemplary of the memory modules 310A-310H. Thebuffered memory module 310 may be of any type such as a SIMM or DIMM,for example. The buffered memory module 310 includes a buffer integratedcircuit chip (“buffer”) 450 and memory integrated circuit chips (“memorydevices”) 452 coupled to a printed circuit board 451. Printed circuitboard 451 includes an edge connector or edge connection 454 that couplesto an edge connector of the host printed circuit board. A southbounddata input (SBDI) and a northbound data output (NBDO) of the memorymodule 310 is respectively received from or transmitted to a priorbuffered memory module or the buffered memory controller. A northbounddata input (NBDI) and a southbound data output (SBDO) of the memorymodule 310 is respectively received from or transmitted to a nextbuffered memory module, if any.

Referring now to both FIGS. 3 and 4, the memory controller 302communicates with the buffers 450 of each memory module 310A-310H ineach memory bank 304 by using the southbound data flow and thenorthbound data blow. The edge connection 454 of the first memory modulebeing the closest to the memory controller in each bank, memory module310A, couples the buffer 450 of each memory module 310A to the memorycontroller 302. Memory module 310A has no adjacent memory module in thenorthbound data flow path. The northbound data flow from memory module310A is coupled to the memory controller 302. The adjacent memory module310A-310H in each bank are coupled together so that data can be read,written, and passed through each buffer 450 of each memory module. Thelast memory module being the furthest from the memory controller in eachbank, memory module 310H, has no adjacent memory module in thesouthbound data flow path. Thus, memory module 310H does not passsouthbound data flow further along the serial chain of memory modules.

The memory controller 302 does not directly couple to the memory devices452 in any memory module. The buffer 450 in each memory module 310A-310Hin each memory bank 304 couples directly to the memory devices 452 onthe printed circuit board 351. The buffer 450 provides data buffering toall the memory integrated circuit chips or devices 452 on the sameprinted circuit board 451 of the memory module 310. The buffer 450further performs serial to parallel conversion and parallel to serialconversion of the data, as well as interleaving/deinterleaving andpacketizing/depacketizing of data as needed. The buffer 450 alsocontrols its portion of the serial chain of the northbound andsouthbound data links with adjacent memory modules. Additionally, in thecase of the first memory module, memory module 310A, the buffer 450 alsocontrols its portion of the serial chain of the northbound andsouthbound data links with the memory controller 302. Additionally, inthe case of the last memory module, memory module 310H, the buffer 450also controls the initialization of the serial chain of memory modulesand the generation of idle frames or idle packets of data in thenorthbound data link and northbound data flow to the memory controller302.

Without a direct coupling between the memory controller 302 and thememory devices 452 of the memory modules, the memory chips or devices452 may be of different types, speeds, sizes, etc. to which the buffer450 may communicate. This allows improved memory chips to be used in amemory module without needing to update the hardware interface betweenthe memory controller and the memory modules by purchasing a new host ormotherboard printed circuit board. The memory module that plugs into thehost or motherboard printed circuit board is updated instead. In oneembodiment of the invention, the memory chips, integrated circuits, ordevices 452 are DDR memory chips with dynamic random access memory(DRAM). Otherwise, in other embodiments of the invention, the memorychips, integrated circuits, or devices 452 can be any other type ofmemory or storage device.

Referring now to FIG. 5, one memory bank 304 of the memory banks304A-304F of a memory system is illustrated in greater detail coupled tothe buffered memory module (BMM) memory controller 302. In oneembodiment of the invention, the BMM memory controller 302 is a fullybuffered dual inline (FBD) memory controller and each of the memorymodules 310A-310H is a fully buffered dual inline (FBD) memory module(FBDIMM). The memory bank 304 includes one or more memory modules310A-310 n daisy chained together. Each memory module 310 functions likea repeater for the valid data flowing in the serial bit streams alongthe northbound data link (NB) and the southbound data link (SB).

Each memory module 310A-310 n in the memory bank 304 includes a buffer450A-450 n, respectively. Each buffered memory module 310A-310Nrespectively includes memory devices 452A-452N which may differ fromeach other. For example, the memory devices 452A in buffered memorymodule 310A may differ from the memory devices 452B in buffered memorymodule 310B. That is, the buffer 450 in each memory module makes thetype of memory used for the memory device transparent from the memorycontroller 302.

The buffer 450 in each memory module functions like a repeater for thedata flowing in the serial bit streams along the northbound data link(NB) and the southbound data link (SB). Additionally, the buffer 450 ineach memory module may insert or merge its own local data into lanes ofserial bit streams flowing along the northbound data link (NB) in placeof frames or partial frames of idle or invalid data.

In order to synchronize the timing of the memory controller 302 and thememory modules 310A-310 n together in the memory bank 304, a clockgenerator 500 is provided that is coupled to each memory module and thememory controller. A clock signal 501 from the clock generator 500 iscoupled to the memory controller 302. Clock signals 502A-502 n arerespectively coupled to the buffers 450A-450 n in memory modules310A-310 n.

Memory controller 302 communicates through the memory modules in thememory bank 304 over the southbound data links SB1-SBn. Memorycontroller 302 may receive data from each memory module 310 within thememory bank 304 over the northbound data links NB1-NBn. The southbounddata links SB1-SBn may consist of one or more lanes of serial data.Similarly, the northbound data links NB1-NBn may consist of one or morelanes of serial data. In one embodiment of the invention, there arefourteen lanes of serial data in the northbound data links NB1-NBn.

The last memory module, memory module 310 n, regardless of whether ornot it has data to send, generates a pseudorandom bit stream and startsit flowing towards the memory controller 302 on the northbound link NBn.The pseudorandom bit stream may be passed from one memory module to thenext on the northbound links NB1-NBn. If the memory module 310 n haslocal data to send to the memory controller 302, it generates a frame ofdata including the local data and places it on the northbound link NBninstead of a frame of data of the pseudorandom bit stream. Thepseudorandom bit stream may include a sequence of bits packetized into aframe of data that indicates an idle frame of data. An idle frame ofdata may be replaced by the other memory modules further down the line(memory modules 310A-3109 n−1) in order to merge a frame of local datainto the serial bit stream flowing on the northbound links NB1-NBn. Forexample, memory module 310B may receive an idle frame on the incomingnorthbound link NB3 and merge a frame of local data in place of the idleframe into the serial bit stream on the outgoing northbound link NB2.

The memory system illustrated in FIG. 5 may further include an SM bus(SMBus) 506 coupled from the memory control 302 to each of the memorymodules 310A-310N. The SM bus 506 may be a serial data bus. The SM Bus506 is a sideband mechanism to access internal registers of the buffer.Certain link parameters may be set up by a BIOS in the buffer beforebrining up the northbound and southbound serial data links. The SM-Busmay also be used to debug the system through access to the internalregisters of the buffer.

The memory controller 302 may be a part of a processor (as illustratedby processor 201′ and memory controller 202′ in FIG. 2B) or may be aseparate integrated circuit (as illustrated by processor 201 and memorycontroller 202 in FIG. 2A). In either case, the memory controller 302can receive write memory instructions with write data from the processorand receive read memory instructions from the processor and supply readdata to the processor in order to respective write or read data to orfrom memory. The memory controller 302 may include a southbound serialoutput interface (SBO) 510 to transmit one or more lanes of serial datato the one or more memory modules in each bank of memory. The memorycontroller 302 may further include a northbound serial input interface(NBI) 511 to receive one or more lanes of serial data from the one ormore memory modules in each bank of memory.

Referring now to FIG. 6 (FIGS. 6-1 and 6-2), a functional block diagramof the buffer 450 for the buffered memory module 310 is illustrated.Buffer 450 is an integrated circuit that can be mounted to the printedcircuit board 451 of the buffered memory module 310. To couple data intoand out of buffered memory module 310, buffer 450 includes a southboundbuffer I/O interface 600A and a northbound buffer I/O interface 600B.

The northbound buffer I/O interface 600B interfaces to the northbounddata out (NBDO) 601 and the northbound data in (NBDI) 602. Thesouthbound buffer I/O interface 600A interfaces to the southbound datain (SBDI) 603 and the southbound data out (SBDO) 604. Northbound data in602 and the northbound data out 601 includes fourteen lanes of a serialdata stream in one embodiment of the invention. The southbound data in603 and the southbound data out 604 includes ten lanes of serial datastreams in one embodiment of the invention.

To interface to the memory devices 452, buffer 450 includes a memory I/Ointerface 612. At the memory I/O interface 612, DRAM data isbidirectionally passed over a DRAM DATA/STROBE bus 605 while addressesand commands are sent out over DRAM ADDRESS/COMMAND buses 606A-606B tothe memory devices. Memory devices 452 are clocked by the DRAM clockbuses 607A-607B in order to synchronize data transfer with the memoryI/O interface 612. From the core logic of the buffer 450, the memory I/Ointerface 612 receives commands over the CMD OUT bus 692 from themultiplexer 635; addresses over the ADD OUT bus 693 from the multiplexer637; and write data over the DATA OUT bus 691 from the multiplexer 636.The write data on the DATA OUT bus 691 is communicated to theappropriate memory devices over the DRAM DATA/STROBE bus 605. Addressdata on the DATA OUT bus 691 is communicated to the appropriate memorydevices over the DRAM ADDRESS/COMMAND buses 606A-606B. The commands onthe CMD OUT bus 692 is communicated to the appropriate memory devicesover the DRAM ADDRESS/COMMAND buses 606A-606B.

In order to generate the core_clock signal 611 for the functional blocksof the buffer 450, it receives a reference clock (REF CLOCK) 502 that iscoupled into a phase lock loop (PLL) 613. The reference clock (REFCLOCK) 502 may be a differential input signal and appropriately receivedby a differential input receiver. Buffer 450 further receives an SM bus506 coupled to an SM bus controller 629. A reset signal (Reset#) 608 iscoupled into a reset control block 628 in order to reset the buffer 450and the functional blocks when it goes active low.

Between the memory I/O interface 612 and the buffer I/O interfaces600A-600B is the core logic of the buffer 450. The core logic of thebuffer 450 is used to read data out from the memory devices and drive itout as local data through the northbound data interface 600B.Additionally, any other response from a memory module is driven out bythe buffer and into the northbound serial data stream through thenorthbound data interface 600B. The core logic of the buffer 450 is alsoused to write data into the memory devices that is received from thesouthbound data interface 600A. The commands to read and write data arereceived from the southbound data interface 600A. If the memory devices452 of the given buffered memory module 310 are not to be accessed,serial data on the northbound data input 602 and the southbound datainput 603 may pass through the buffer I/O interface 600A-600B onto thenorthbound data output 601 and the southbound data output 604,respectively. In this manner, data from another buffered memory module310 will be passed through to the memory controller on the northbounddata interface 600B without having to be processed by the core logic ofthe buffer 450. Similarly, data from the memory controller may be passedon to another memory module on the southbound data interface 600Awithout having to be processed by the core logic of the buffer 450.

The core logic of the buffer 450 includes functional blocks to read datafrom and write data into the memory devices 452. The core logic of thebuffer 450 includes a phase lock loop (PLL) 613, a data CRC generator614, a read FIFO buffer 6633, a five into one bus multiplexer 616, async and idle pattern generator 618, a NB LAI buffer 620, an integratedbuilt in self-tester for the link (IBIST) 622B, a link initialization SMand control and configuration status registers (CSRs) 624B, a resetcontroller 625, a core control and configuration status registers (CSRs)block 627, an LAI controller block 628, an SMbus controller 629, anexternal MEMBIST memory calibration block 630, and a failover block 646Bcoupled together as shown in FIG. 6. The core logic of the buffer 450may further include a command decoder and CRC checker block 626, an idlebuilt in self-tester (IBIST) block 622A, a link initialization SM andcontrol and CSRs block 624A, a memory state controller and CSRs 632, awrite data FIFO buffer 634, a four into one bus multiplexer 635, a fourinto one bus multiplexer 636, a three into one bus multiplexer 637, anLAI logic block 638, an initialization patterns block 640, a two intoone bus multiplexer 642, and a failover block 646A coupled together asshown in FIG. 6.

A multiplexer includes at least two data inputs, an output, and at leastone control or select input to select the data input that is to beprovided at the output of the multiplexer. For a two input multiplexer,one control or select input is used to select the data that is output atthe multiplexer. A bus multiplexer receives a plurality of bits at eachdata input and has an output with a plurality of bits as well. A twointo one bus multiplexer has two buses as its data input and a singlebus output. A three into one bus multiplexer has three buses as its datainput and a single bus output. A four into one bus multiplexer has fourbuses as its data input and a single bus output.

Within the buffer 450, each of the buffer I/O interfaces 600A-600Bincludes a FIFO buffer 651, data merge logic 650, a transmitter 652, areceiver 654, a re-synchronization block 653, and a demultiplexer/serialparallel converter block 656. Data can pass through each of the bufferI/O interfaces 600A-600B through a resynchronization path 661 ora/re-sample path 662 without interfacing to the core logic. Through theembodiments of the invention, local data associated with the buffer 450can be merged into the serial data stream to overwrite an idle framewithout having the core logic receive a serial data stream and determinewhere the idle frames are located there-in.

The multiplexer 616 selects what data is directed towards the FIFObuffer 651 of the northbound buffer I/O interface 600B for driving outas local data on the serial lanes of the northbound data output 601.Generally, the multiplexer 616 may select status or other controlinformation from the core control and CSR block 627, read data from theread FIFO buffer 633, read data with attached CRC data from the CRCgenerator 614, synchronization or idle patterns from the patterngenerator 618, or test pattern data from the IBIST block 622B.

The multiplexer 642 selects what data is directed towards the FIFObuffer 651 of the southbound buffer I/O interface 600A for driving outon the serial lanes of the southbound data output 604. Generally, themultiplexer 642 may select initialization patterns from the initpatterns block 640 or test pattern data from the IBIST block 622A.

Referring now to FIG. 7A, a block diagram of the data merge logic 650coupled to the transmitter 652 is illustrated. The transmitter 652 ismade up of N lanes of transmitters 752A-752 n. As discussed previously,in one embodiment of the invention the number of lanes is ten. Inanother embodiment of the invention, the number of lanes is fourteen. Inthe data merge logic 650 there is a data merge logic slice 700A-700 nfor each one of the N lanes.

A parallel local data bus 660 from the first-in-first-out (FIFO) buffer651 couples into each data merge logic slice 700A-700 n. Respectivelanes of serial data of the re-synch bus 661 couple into each respectivedata merge logic slice 700A-700 n. The bit width of the re-synch bus 661is two times the number of lanes. Two bits of each respective lane ofthe re-synch bus 661 are coupled into each respective data merge logicslice 700A-700N. Respective lanes of serial data of the re-sample bus662 couple into each respective data merge logic slice 700A-700 n. Thebit width of the re-sample bus 662 is two times the number of lanes. Twobits of each respective lane of the re-sample bus 662 are coupled intoeach respective data merge logic slice 700A-700N.

The re-sample bus 662 and the resynch bus 661 both transfer a two bitserial data stream for each lane into each respective data merge logicslice 700A-700N. In contrast, the parallel data bus 660 couples six ortwelve bits for each lane into each respective data merge logic slice700A-700N. The bit width of the parallel local data bus 660 is twelvetimes the number of lanes. However in a six-bit mode, only six bits ofthe twelve may be active per lane. The output from each of the datamerge logic slices 700A-700N is a two bit serial data stream which isrespectively coupled into the serial transmitters 752A-752N. Each serialtransmitter 752 converts two parallel bits of serial data into a singlebit serial data stream on the respective lane 601A-601N of thenorthbound data output (NBDO) 604 or the respective lane 604A-604N ofthe southbound data output (SBDO) 601 as shown in FIG. 7A.

Referring now to FIG. 7B, a schematic diagram of a data merge logicslice 700 i is illustrated coupled to a transmitter 752 i. The datamerge logic slice 700 i represents one of the data merge logic slices700A-700 n for each of the N lanes illustrated in FIG. 7A. Thetransmitter 752 i represents one of the transmitters 752A-752 n for eachof the N lanes illustrated in FIG. 7A.

Each data merge logic slice 700 i can operate in one of two bit widthmodes, a full frame mode of twelve bits width (also referred to as a 12bit mode) or a half frame mode of six bits width (also referred to as asix-bit mode). A mode control signal (6bit_mode) 722 indicates andcontrols which of the two bit width modes the data merge logic slice 700i is to function with the core logic.

In the full frame mode or twelve bit mode, the core logic uses a fullframe of twelve bits to communicate data over bus 660 i with the datamerge logic slice 700 i. The lower six bits of bus 660 i are representedby the Data[5:0] bus 726 while the upper six bits of bus 660 i arerepresented by a Delayed_data[5:0] bus 727. The twelve bits of localdata (Data[5:0] and Delayed_data[5:0]) that are to be merged into theserial data stream and transmitted are respectively latched into a lowerparallel-in-serial-output (PISO) converter 708B and an upperparallel-in-serial-output (PISO) converter 708A at the beginning of theframe by an “Early_Load_Pulse” control signal 720.

The lower parallel-in-serial-output (PISO) converter 708B and the upperparallel-in-serial-output (PISO) converter 708A areparallel-in-serial-output (PISO) shift registers and may be also bereferred to herein as such. Each of the PISO converters 708A-708B, alsoreferred to as PISO shift registers 708A-708B, have a parallel datainput, a clock input, a load/shift-bar input, a serial input (SIN), anda serial output (SO). The serial output of the upper PISO shift register708A is coupled into the serial input of the lower PISO shift register708B to support serializing twelve parallel bits of the local data bus660 i. The serial input of the upper PISO shift register 708A may becoupled to a logical low (e.g., ground) in one embodiment of theinvention or a logical high (e.g., VDD) in another embodiment of theinvention. The serial output (SOUT) of the PISO shift registers708A-708B is two bits at a time in one embodiment of the invention. Inanother embodiment of the invention, the serial output (SOUT) of thePISO shift registers 708A-708B may be one bit at a time.

In the twelve bit mode, the six bits of bus 726 are coupled to theparallel data input (PIN) of the lower PISO shift register 708B whilethe six bits of bus 727 are coupled to the parallel data input (PIN) ofthe upper PISO shift register 708A. These twelve bits are respectivelyloaded into each PISO shift register during the early load pulse 720with the mode control signal 722 indicating a twelve bit bus mode (e.g.,mode control signal 722 indicates twelve bit mode by being a logical lowlevel and a six-bit mode by being a logical high level in one embodimentof the invention). In the twelve bit mode, the clear input to D-typeflip flop 706A is logically high setting the Q output of the D-type flipflop 706A to logical zero so that the control input to the multiplexer703 selects the bus 726 to be output onto bus 728.

In the half frame mode or six-bit mode, the core logic only uses a halfframe of six bits to communicate data over bus 660 i with the data mergelogic slice 700 i at a time. The core logic sends six bits of data at atime or early data (Data[5:0]726) and late data (Delayed_data[5:0])offset by half of a frame. In the half frame mode, only the lower PISOshift register 708B of the data merge logic slice 700 i is used to mergedata into the serial data stream for transmission.

In the six-bit mode, the multiplexer 703 selectively couples the sixbits of bus 726 to the parallel data input (PIN) of the lower PISO shiftregister 708B during the early load pulse 720 and the six bits of bus727 to the parallel data input (PIN) of the lower PISO shift register708B during the late load pulse 721. The six bits of bus 726 are loadedinto the PISO shift register 708B during the late load pulse 721 withthe mode control signal 722 indicating a six-bit bus mode. The six bitsof bus 727 are loaded into the PISO shift register 708B during the earlyload pulse 720 with the mode control signal 722 indicating a six-bit busmode.

The data merge slice 700 i includes data path logic and control logic701 i. The data path logic selectively merges the local data and thefeed-through data into the serial bit stream. The control logic 701 icontrols the data path logic in each data merge slice in order toproperly synchronize the merging of local data and feed-through datainto the serial bit stream.

The control logic 701 i, with mode control logic and merge controllogic, includes three single bit two to one multiplexers 702A-702C,set/reset D flip-flops 706A-706B, an OR gate 710, an AND gate 711, andan inverter 712 coupled together as shown and illustrated in FIG. 7B.The signals generated by the control logic 701 i are coupled to the datapath logic. The multiplexers 702A-702B, the D-type flip flop 706A, theOR gate 710, the AND gate 711, and the inverter 712 provides modecontrol logic. The multiplexer 702C and the D-type flip flop 706Bprovide merge control logic.

The data path logic includes a six-bit two to one bus multiplexer 703,two bit two-to-one bus multiplexers 704-705, and a pair of six-bitin/two bit out parallel in serial out (PISO) converters 708A-708Bcoupled together as shown and illustrated in FIG. 7B.

Each slice 700 i of the data merge logic 650 may receive a two bitserial lane of resynch data 661 i, a two bit serial lane of re-sampledata 662 i, and a twelve bit parallel lane of local data 660 i. Theparallel lane of local data 660 i is from the core logic of the buffer450 and may be various types of data. For example, the local data 660 imay be read data from memory devices 452, cyclic redundancy check (CRC)data, test data, status data, or any other data that is to be received,transmitted, or generated by the core logic of the buffer.

The two bit lane of re-sync data 661 i and the two bit lane of re-sampledata 662 i have no contact with the core logic of the given buffer 450and are multiplexed into feed-through data (also referred to herein as“feedthru data”) 725 by multiplexer 705 in response to a local clockmode signal 736. If the buffer 450 is operating in the local clock mode,the resynch data is multiplexed onto the feedthru data 725. If thebuffer 450 is not operating in the local clock mode, the re-sample data662 i is multiplexed onto the feedthru data 725. In a local clock mode,a phase locked loop (PLL) clock generator is used to generate a localclock signal in the buffer that is used to resynchronize the inputserial data stream to generate the re-synch data. If not in the localclock mode, a received clock is generated from and synchronized with theframes of data in the received serial data stream that is used to samplethe input serial data stream to generate the re-sample data. Theclock2UI signal 723 is switched between the locally generated clocksignal and the received clock signal in response to the local clock modesignal 736. The source of the feedthru data 725 may be from the buffer450 of another memory module 310 on the northbound (NB) side (alsoreferred to as forwarded northbound data); or from the buffer 450 ofanother memory module 310 on the southbound (SB) side (also referred toas forwarded southbound data) or alternatively from the memorycontroller 302 on the southbound (SB) side.

The two to one bus multiplexer 704 receives the two bits of serialfeedthru data 725 as a first input, a two bit serial output from the 6-2PISO shift register 708B as a second input, and a local data selectsignal (PISO_SEL) 732 at its control input. The two bit serial output735 from the 6-2 PISO shift register 708B is two serialized bits oflocal data 735 from the parallel data bus 660 i. Thus in response to thelocal data select signal (PISO_SEL) 732, the multiplexer 704 eitherselects to output two bits of feedthru data 725 or two bits ofserialized local data 735 from the parallel data bus 660 i generated bythe 6-2 PISO shift register 708B. The two bit output 730 from themultiplexer 704 is coupled to the transmitter 752 and further serializedinto a single bit onto the lane NBDOi/SBDOi 601 i,604 i. In this manner,local data from the core logic can be multiplexed with feed-through dataand merged into a lane of the serial bit stream at NBDOi/SBDOi 601 i,604i.

The local data select signal (PISO_SEL) 732 that controls the merging ofdata into the serial bit stream is generated by the D flip-flop 706B. Inresponse to a merge enable signal 724, the D flip-flop 706B generatesthe local data select signal (PISO_SEL) 732 on the rising edge of theclock signal Clock_(—)2UI 723. The merge enable signal 724 is coupledinto a first input of the multiplexer 702C. The local data select signal(PISO_SEL) 732 is fed back and coupled into a second input of themultiplexer 702C. The output of multiplexer 702C is coupled into the Dinput of the D flip-flop 706B. An early load pulse (EARLY_LD_PULSE)signal 720 is coupled into the select control input of the multiplexer702C. If the early load pulse 720 is active high, the merge enablesignal 724 is driven out by the multiplexer 702C and coupled into the Dinput of the D flip-flop 706B. If the early load pulse 720 is low, thelocal data select signal (PISO_SEL) 732 is fed back through themultiplexer 702C and coupled into the D input of D flip-flop 706B toretain the current state of the local data select signal (PISO_SEL) 732.As the early load pulse 720 is periodically clocked, if the merge enablesignal 724 is low, it clears the D flip flop 706B so its Q output is alow logic level signal that terminates the merge of data at theappropriate time.

The merge enable signal 724 is synchronized into the local data selectsignal (PISO_SEL) 732 on the edge of the clock signal Clock_(—)2UI 723.As the merge_enable signal 724 is sampled during the early_load_pulse720 to generate the local data select signal (PISO_SEL) 732, themultiplexer 704 is switched on frame boundaries (12 bits of data perlane in a frame). If the merge enable signal 724 is active high on therising edge of the clock signal Clock_(—)2UI 723, the local data selectsignal (PISO_SEL) 732 goes active high to control the multiplexer 704 toselect the two serialized bits of local data 735 as its two bit output730. If the merge enable signal 724 is low on the rising edge of theclock signal Clock_(—)2UI 723, the local data select signal (PISO_SEL)732 remains low to control the multiplexer 704 to select the twofeedthru bits of data 725 as its two bit output 730.

In response to the local data select signal (PISO_SEL) 732 being alogical high, the two serial bits in the parallel data bus 660 i are tobe merged into the lane NBDOi/SBDOi 601 i,604 i. In response to thelocal data select signal (PISO_SEL) 732 being a logical low, the twobits of feedthru data 725 is selected by multiplexer 704 to be outputonto the lane NBDOi/SBDOi 601 i,604 i.

As the local data select signal (PISO_SEL) 732 is responsive to themerge enable signal 724, the generation of the merge enable signal 724allows the parallel data of bus 660 i to be merged onto the serial datastream of the lane NBDOi/SBDOi 601 i,604 i. The merge enable signal 724is generated by link control logic (in the link init SM and control andCSR functional block 624B illustrated in FIG. 6) in time to allow localdata to be merged into the serial data stream at the appropriate time.

Referring momentarily back to FIG. 5, the timing of the merge enablesignal is established for each memory module 310 during initializationand training of the system. Note that for the last memory module 310 nin a bank 304, the merge enable signal is more of a data transmit signalas there are no further memory modules in the chain generating data inthe northbound data link.

Referring now to FIG. 10, a flow chart is illustrated for theinitialization, training, and functioning of the buffer in merging localdata and feed through data together into a serial data stream output.The flow chart starts at block 1000.

At block 1002, the buffer in each memory module of each memory bank isinitialized. During initialization of a memory bank 304, each memorymodule has its southbound and northbound serial data links initialized(may also be referred to as being part of link training). The memorycontroller 302 sends out an initialization pattern on the southbound(SB) data link SB1-SBn. During initialization, the buffer 450 n in thelast memory module 310 n receives the initialization pattern on thesouthbound data link SBn and retransmits it back onto the northbound(NB) data link NB1-NBn through other memory modules back to the memorycontroller 302. As each buffer has its own clock, the initializationpattern received on the northbound (NB) data link NB1-NBn by the bufferis used for bit locking and frame alignment purposes in each lane ofserial data. The clock in the buffer may be synchronized to theinitialization pattern. The timing of logic may be aligned with theinitialization pattern in order to receive packets of data in the serialdata stream as well as parse a header from a frame of data and any errorcorrection/detection or other data fields within a packet. Thegeneration of the Early_Ld_Pulse 720 is set up to be coincident with thebeginning of frames of data received by a given memory module. Thegeneration of the Late_LD_Pulse 721 is set up to be at a half frameboundary of frames of data received by a given memory module.

Next at block 1004, each buffer in each memory module of each memorybank is trained. After sending out the initialization pattern, thememory controller 302 sends out a training pattern through to the lastmemory module 310 n in a given bank 304 during training. Duringtraining, the buffer 450 n in the last memory module 310 n receives thetraining pattern on the southbound data link SBn and retransmits it backonto the northbound (NB) data link NB1-NBn through other memory modulesback to the memory controller 302. Each memory module observes one ofthe training patterns on the southbound (SB) data link and determinesthe amount of time or clock cycles for it to return to the same memorymodule on the northbound (NB) data link. A roundtrip time is determinedfor a given position of each memory module.

Provided that the requests are not overly bunched together, theroundtrip time represents a slot in time where it is safe for a givenmemory module to merge data onto the northbound data link withoutcolliding with valid data of another memory module. At a given memorymodule, an idle data packet is expected to be received at this point intime on the northbound data link after seeing a memory request commandon the southbound data link. At this point in time, the idle data packetcan be replaced by a local data packet. The roundtrip time and thecommand to data delay time for a given memory module are the basis forsetting up the timing of the merge enable signal that is used to controlthe merging of local data into the northbound data link. If theroundtrip time is long, data can be fetched in advanced and placed in aFIFO buffer waiting for the proper moment to be merged into thenorthbound data stream. The distance between read and write FIFO bufferpointers in the northbound interface of the buffer can be set based onthe roundtrip timing. The roundtrip time may be determined as a functionof a whole number of periods of the bit rate clock, clock_(—)2UI 723.The number of memory modules in a channel and the command to data delayof the last memory module in the channel determines the round-trip timefor that channel.

A command to data delay for each memory module may be further determinedto assist in establishing the timing of the merge enable signal in eachmemory module. The command to data delay timing may include one or moreof the following time periods: the time for a command to be transferredfrom the southbound IO interface 600A to the memory IO interface 612;the time for the command to be transferred from the memory IO interface612 to the memory devices 452; differences in clock timing for thememory IO interface 612 and the memory devices 452, routing delays inthe clock signals and command signals to the memory devices 452; anyset-up/hold times for the buffer 450 and the memory devices 452; theread latency in the memory devices 452 (e.g., CAS timing and any addedlatency); routing delays in the data signals and strobe signals from thememory devices 452 to the buffer 450; data delay skew between memorydevices; delays through the memory IO interface 612, any set-up/holdtimes for the buffer 450 and the memory devices 452; and time for datato be transferred from the memory IO interface 612 to the northbound IOinterface 600B (this may include buffering and clocking delays for datawithin the buffer 450). The command to data delay timing may bedetermined as a number of multiple of frames or a fraction there-of witha granularity of the delay time being as a function of a whole number ofperiods (bit times such as frame/12 or clock_(—)2ui/2) of a bit rateclock. The command to data delay timing of a memory module, such as thelast memory module 310 n, can be programmatically increased by aregister setting if additional delay time is desired.

Next at block 1006, after the initialization and training, each bufferis ready to receive an input serial data stream from a serial datainput. However, the buffer in the last memory module 310 n in the memorybank 304 either transmits idle packets or read requested data packets onthe northbound data link towards the memory controller 302. Otherwise,an input serial data stream is received that represents feed-throughframes of data interspersed between idle frames of data.

Next at block 1008, a determination may be made with respect to theavailability of local data. If there is local data to merge into theserial data stream, then the control flow jumps to block 1010. If thereis no local data to merge into the serial data stream, then the controlflow jumps to block 1014.

At block 1014 with no local data to merge, the feed-through data istransmitted onto the serial data output. The feed-through data may haveits bits of data in the input serial data stream re-sampled.Alternatively, feed-through data may have its bits of data in the inputserial data stream resynchronized. Then the control flow jumps back togo to block 1006 to continuously receive the input serial data stream.

At block 1010 with local data to merge, frames of the local data replacethe feed-through data in the output serial data stream. That is, iflocal data needs to be sent by a buffer, frames of data in the incomingserial data stream are dropped and frames of local data are sent inplace thereof in response to the merge enable signal. The frames of thelocal data and the feed-through data may be merged together byserializing parallel bits of the local frames of data into serial bitsof data and then multiplexing the serial bits of data of the localframes of data and the serial bits of the feed-through frames of datainto serial bits of the output serial data stream in response to themerge enable signal. During initialization and training, the host andmemory controller ensures that idle frames of data in the input serialdata stream are replaced by local frames of data. The buffer does notneed to check if the incoming frame in the input serial data stream thatis being replaced is an idle frame of data or not.

At block 1012, the output serial data stream, including the merged data,is transmitted onto the serial data output to the next memory module upthe chain or alternatively to the memory controller.

Next, the control process jumps back to block 1006 to continue receivingthe input serial data stream from the serial data input.

As discussed previously, the local data from the core logic and thebuffer 450 may be output in six bits, chunks or twelve bit chunks at atime. The mode control signal (6bit_mode) 722 determines whether thedata merge logic slice 700 i is to function in a six-bit mode (halfframe mode) or a twelve bit mode (full frame mode). The mode controlsignal (6bit_mode) 722 is coupled into the selection or control input ofthe multiplexer 702A and a first input of the AND gate 711, and theinput to the inverter 712.

The early load pulse signal 720 controls the loading of the first sixbits on the parallel data bus 660 i. A late load pulse signal 721controls the loading of the second six bits on the parallel data bus 660i. A late load pulse 721 is coupled into a first input of the OR gate710. The early load pulse control signal 720 is coupled into the firstinput of the multiplexer 702B, the second input of the OR gate 710, thefirst input of the multiplexer signal 702A, a load/shift-bar input ofthe 6-2 PISO shift register 708A, and the select input of themultiplexer 702C.

The clock signal Clock_(—)2UI 723 couples into the clock inputs of the Dflip-flops 706A-706B, and the clock inputs of the 6-2 PISO shiftregisters 708A-708B. The output of multiplexer 702A is coupled into theload/shift-bar input of the 6-2 PISO shift register 708B.

The parallel input of the 6-2 PISO shift register 708A is coupled to thesix bit delayed data bus 727. The two bit serial output of the 6-2 PISOshift register 708A is coupled into the two bit serial input of the 6-2PISO shift register 708B. The parallel input of the 6-2 PISO shiftregister 708B is coupled to the six-bit output from the multiplexer 703.In this manner when a data merge logical slice 700 i is in a 12 bitmode, 12 bits of data can be loaded into the 6-2 PISO shift register708A-708B and then shifted serially out from the 2 bit serial output708B, through the multiplexer 704 and coupled into the transmitter 752i.

The serial transmitter 752 i is double clocked by a clock signal inorder to convert the 2 parallel bit sets into the serial single bit atits output 601 i, 604 i.

The data merge logical slice 700 i is in a 12 bit mode when the6bit_mode control signal 722 is a logical low. The data mergelogical/700 i is in a 6 bit mode when the 6 bit mode controls signal 722is a logical high. Control logic 710-712 in conjunction with themultiplexer 702B and D flip-flop 706A generate a data bus select(Data_Sel) signal 729 which is coupled to the select input of themultiplexer 703 in order to establish a 12 bit mode or a 6 bit mode inresponse to the 6 bit mode control signal 722. When the data bus selectsignal 729 is logically low, 12 bits of data are to be loaded inparallel into the 6-2 PISO shift registers 708A-708B. When the data busselect signal 729 is a logical high, 6 bits of the data bus 727 are tobe coupled into the 6-2 PISO shift register 708B.

In a 6 bit mode, either the early load pulse signal 720 or the late loadpulse 721 can load parallel data into the 6-2 PISO shift register 708B.In either the 6 bit or 12 bit mode, the early load pulse 720 is onlyused to load parallel data from the data bus 727 into the 6-2 PISO shiftregister 708A.

The serial input of the 6-2 PISO shift register 708A is coupled toground such that only zeros will be serially shifted in behind the datato be transmitted. Alternatively, the serial input of the 6-2 PISO shiftregister 708A may be connected to VDD such that only logical ones areserial shifted in behind the data being transmitted.

The Q output of D flip-flop 706A is coupled into the second input of themultiplexer 702B such that when the output of AND gate 711 is a logicalrow, the Q output couples into the D input of the D flip-flop 706A toretain the loaded logic state therein of the data bus select (DATA_SEL)signal 729.

Referring now to FIG. 8, a timing diagram of waveforms depicting thedata merge logic slice 700 i functioning in a twelve-bit mode isillustrated. That is, the 6 bit mode control signal 722 is a logical lowin the timing diagram of FIG. 8.

In FIG. 8, Clock_(—)2UI signal 723 is illustrated by the waveform 823.The core clock signal 611 is illustrated by the waveform 811. The lowersix bits of data (MEM_DATA IN [5:0]) 690A on the parallel data bus 690is illustrated by the waveform 890A. The upper six bits of data(MEM_DATA IN [11:6]) 690B on the parallel data bus 690 is illustrated bythe waveform diagram 890B. The lower six bits of data (FBD_DATA [5:0])726 on the parallel data bus 660 i is illustrated by the waveformdiagram 826. The upper six bits of data (FBD_DATA [11:6]) 727 on theparallel data bus 660 i is illustrated by the waveform diagram 827. Themerge enable control signal 724 is illustrated by the waveform diagram824. The early load pulse control signal 720 is illustrated by thewaveform 820. The late load pulse control signal 721 is illustrated bythe waveform 821. The local data select control signal (PISO_SEL) 732 isillustrated by the waveform 832. The single bit serial output datastream NBDOi 601 i is illustrated by the waveform 801.

Without any local data to merge into the northbound serial data stream,the buffer 450 passes the received bits on the northbound data input 602(“Feedthru Data” 725) to the transmitter 752 i in the high speed clockdomain, bypassing the core logic of the buffer 450. The local dataselect control signal (PISO_SEL) 732 is low when the Feedthru Data 725is multiplexed into the transmitter 752 i as is illustrated by thewaveform 832.

As discussed previously, the “Early_Ld_Pulse” 720 is set up to becoincident with the beginning of a frame (as seen on the link) and thelate_ld_pulse 721 is set up to be at half frame boundary during theinitial training of a lane of the serial data link. A frame of data is alogical unit of data over the link when in a full frame operational modeand is made up of twelve bits of data in one embodiment of theinvention.

In full frame operational mode, twelve bits of a frame are loaded intothe PISO shift register using the “Early_Ld_Pulse” signal 720. The“late_ld_pulse” signal 721 is not used to load bits into the PISO shiftregisters. Both the upper and lower PISO shift registers 708A-708B areused in this mode. The six bit_mode control signal 722, being low in thetwelve bit mode, causes the Data_Sel” signal 729 to be low in twelve bitmode by clearing the output of the D flip flop 706A. With the “Data_Sel”signal 729 being low in the twelve bit mode, the six lower data bits(FBD_DATA[5:0] 726 of bus 660 i are coupled into the lower PISO Shiftregister 708B through the multiplexer 703.

The periodic generation of the Early_Ld_Pulse 720 also enables samplingof the “Merge_enable” signal 724 by the D flip-flop 706B. The periodicgeneration of the Early_Ld_Pulse 720, being active high, selectivelycontrols the multiplexer 702C to select the merge_enable signal 724 asits output data that is coupled into the data input D of the D flip-flop706B.

As discussed previously, the merge enable signal 724 is generated at anappropriate time to insert local data from a given memory module into alane of northbound serial data, replacing an idle frame or packet ofdata in the serial data stream. Waveform 824 illustrates an active highpulse 844 being generated when local data is made available on the upperbits (FBD_DATA [11:6]) 727 and lower bits (FBD_DATA [5:0]) 726 of thedata bus 660 i.

When the active high pulse 844 is generated in the waveform 824 of themerge enable signal 724, the pulses 840A-840B in the early_ld_pulsesignal 720 allow the active high pulse 844 of the merge enable signal724 to be sampled by the D flip-flop 706B using the clock_(—)2UI signal723. This causes an active high pulse 842 to be generated in thewaveform 832 of the local data select signal (PISO_SEL) 732. The activehigh pulse 842 of the local data select signal (PISO_SEL) 732 causes themultiplexer 704 to switch from providing the two-bit “Feedthru Data” 725at its output to provide the two-bit serialized local data 735 at itsoutput instead. The switch from feed-through data 725 to local data 735occurs at the frame boundary when the active high pulse 842 is firstgenerated. This is because the falling edge of the “Early_Ld_Pulse” 720that allows the PISO shift registers 708A-708B to start shifting iscoincident with the frame starting point.

When merging data with the “Early_Ld_Pulse” 720 and the multiplexeroutput 731 both low, the PISO shift registers 708A-708B serially shiftout the twelve bits of local data two bits at a time on the serialoutput 735 using the “Clock_(—)2ui” clock signal 723. The transmitter725 i further serializes the two bits into a single bit serial datastream on the NBDOi output 601 i as illustrated by the local dataindicated above the waveform 801.

Referring now to FIG. 9, a timing diagram of waveforms depicting thedata merge logic slice 700 i functioning in a six-bit mode isillustrated. That is, the six-bit mode control signal (6BIT_MODE) 722 isa logical high as illustrated by the waveform 922 in the timing diagramof FIG. 9.

In FIG. 9, the Clock_(—)2UI signal 723 is illustrated by the waveform923. The core clock signal (core_clk) 611 is illustrated by the waveform901. The lower six parallel data bits (MEM_DATA IN [5:0]) 690A on thememory data bus 690 are illustrated by the waveform 990A. The upper sixparallel data bits (MEM_DATA IN [11:6]) 690B of the memory data bus 690are illustrated by the waveform 990B. The lower six bits of data(FBD_DATA [5:0]) 726 on the parallel data bus 660 i are illustrated bythe waveform diagram 926. The upper six bits of data (FBD_DATA [11:6])727 on the parallel data bus 660 i, are illustrated by the waveformdiagram 927. The merge enable control signal 724 is illustrated by thewaveform diagram 924 that occurs earlier than that of the waveform 824in FIG. 8. The early load pulse control signal (EARLY_LD_PULSE) 720 isillustrated by the waveform 920. The late load pulse control signal(LATE_LD_PULSE) 721 is illustrated by the waveform 921. The data busselect control signal (DATA_SEL) 729 is illustrated by the waveform 929.The local data select control signal (PISO_SEL) 732 is illustrated bythe waveform 932. The single bit serial output data stream NBDOi 601 iis illustrated by the waveform 901.

In the six-bit mode, the lower PISO shift register 708B is used toconvert parallel bits of data into serial data by shifting bits out. Thedata bus select signal (DATA_SEL) 729 toggles whether the leastsignificant six bits of the frame, FBD_Data[5:0] 726, or the mostsignificant six bits of the frame, FBD_Data[1:6] 727, are loaded intothe lower PISO shift register 708B through the selected output of thebus multiplexer 703.

Both of the “Early_Ld_Pulse” 720 and the “Late_Ld_Pulse” 721 can causethe lower PISO shift register 708B to either load data or shift data outbecause the output of the OR gate 710 is coupled into the load/shift-barinput of the lower PISO shift register 708B through the multiplexer 702Awhen the 6BIT_MODE signal 722 is active high.

When the “Early_Ld_Pulse” 720 and the “Late_Ld_Pulse” 721 are low, bitsare shifted out from the lower PISO shift register 708B. Also during theparallel load of bits into the lower PISO shift register 708B when theload/shift-bar control input is high, bits previously loaded continue tobe shifted out. When the load/shift-bar control input returns to lowafter a parallel load of data bits, the newly loaded bits are thenshifted out by the lower PISO shift register 708B. In this manner, allsix bits of data may be shifted out while a new set of parallel bits isbeing loaded.

The least significant six bits of the frame, FBD_Data[5:0] 726, areloaded into the lower PISO shift register 708B by the pulses 940A and940B in waveform 920 of the “Early_Ld_Pulse” 720 when the data busselect signal (DATA_SEL) 729 is low, such as at low points 949C,949D forexample. The most significant six bits of the frame, FBD_Data[11:6] 727,are loaded into the lower PISO shift register 708B by the pulses 941Aand 941B in waveform 921 of the “Late_Ld_Pulse” 721 when the data busselect signal (DATA_SEL) 729 is high, during pulses 949A,949B forexample.

In the six-bit mode, the switching between serialized “Feedthru_Data”725 and the serialized local data 735 is similar to the twelve bit modeof operation described previously and is not repeated here for reasonsof brevity.

When merging data, the PISO shift register 708B alternates betweenserially shifting out six most significant bits or six least significantbits of local data two bits at a time onto the serial output 735 usingthe Clock_(—)2UI clock signal 723. The transmitter 725 i furtherserializes the two bits into a single bit serial data stream on theNBDOi output 601 i as illustrated by the local data indicated above thewaveform 901.

While in a six-bit mode a full frame of data is still being transmitted,embodiments of the invention further reduce the latency of local databeing merged into the serial data stream. In comparing FIGS. 8 and 9together, the merging of local data occurs one frame time earlier inFIG. 9.

Embodiments of the invention enable merging of feed-through data andlocal data together into the serial data link on-the-fly without havingto decode incoming packets of the serial input data stream to determinethe location of an idle packet. Previously, the incoming serial datastream was received, depacketized/decoded, and reassembled into framesby the core logic before being re-transmitted. Embodiments of theinvention avoid the depacketizing/decoding of the incoming serial datastream and its reassembly into frames of data and thenencoding/packetizing for retransmission. The embodiments of theinvention enable re-transmission of the incoming serial data stream andthe merging of local data into the serial data stream without involvingthe core logic of a buffer integrated circuit. In a multi memory modulesystem, the serial communication channels may continue to function eventhough a memory integrated circuit in one of the daisy chained memorymodules is non-functional.

Embodiments of the invention are designed to provide for low latencymemory access operations. This can allow a larger memory with morememory modules to be provided in each bank without memory access latencydegrading the system performance as the number of memory modulesincrease in a channel.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of and not restrictive on the broad invention, andthat this invention not be limited to the specific constructions andarrangements shown and described, since various other modifications mayoccur to those ordinarily skilled in the art. For example, oneembodiment of the invention has been described to provide a serial datalink for a fully buffered dual inline memory module. However,embodiments of the invention may be implemented in other types of memorymodules and systems. As another example, data was serialized two bits ata time on two bit buses around the PISO shift registers 708A-708B withinthe merge logic to provide relaxed data timing in one embodiment of theinvention. However, embodiments of the invention may use a single bitoutput PISO with different clock timing and serialize the local datainto a single bit serial data stream with the feed through data andmultiplexers 704,705 being provided to support a single bit serial datastream.

1. An integrated circuit comprising: a serial input/output interfacewith one or more lanes, each of the one or more lanes including a firstparallel-in-serial-output (PISO) shift register having a parallel inputcoupled to a local data bus, a clock input coupled to a first clocksignal, a load/shift-bar input coupled to a load signal, the first PISOshift register to serialize parallel data on the local data bus intoserialized local data on a first serial output; a first multiplexerhaving a first data input coupled to the first serial output, a seconddata input to receive feed-through data, a select input coupled to alocal data select signal, the first multiplexer to selectively merge theserialized local data and the feed-through data into a serial datastream on a multiplexed output responsive to the local data selectsignal; and a transmitter having an input coupled to the multiplexedoutput to receive the serial data stream, the transmitter to drive theserial data stream onto a serial data link.
 2. The integrated circuit ofclaim 1, wherein: the feed-through data is two bits wide: the parallelinput of the first PISO shift register is at least six bits wide and theserial output of the first PISO shift register is two bits wide: thefirst multiplexer is a two bit bus multiplexer and the serial datastream at the multiplexed output is two bits wide: and the transmitterreceives the two bit serial data stream and serializes it onto theserial data link as a single bit serial data stream.
 3. (canceled) 4.The integrated circuit of claim 1, wherein each of the one or more lanesfurther includes control logic coupled to the first multiplexer and thefirst PISO shift register, the control logic to receive the first clocksignal and a merge enable signal, the control logic to generate thelocal data select signal to merge the serialized local data and thefeed-through data into the serial data stream in response to the mergeenable signal and the first clock signal.
 5. The integrated circuit ofclaim 4, wherein the control logic has merge control logic thatincludes: a second multiplexer having a first data input coupled to themerge enable signal and a select input coupled to the load signal: aD-type flip-flop coupled to the second multiplexer, the D-type flip-flophaving a data input coupled to an output of the second multiplexer, anda clock input coupled to the first clock signal, and a data outputcoupled to the select input of the first multiplexer and a second datainput of the second multiplexer, the D-type flip-flop to register themerge enable signal in response to the load signal and the first clocksignal to generate the local data select signal on the data output: andwherein the second multiplexer to recirculate the local data selectsignal into the data input of the D-type flip-flop in response to aninverse of the load signal.
 6. (canceled)
 7. The integrated circuit ofclaim 1, wherein each of the one or more lanes further includes a busmultiplexer coupled between the local data bus and the first PISO shiftregister, the bus multiplexer having a first data input coupled to theone half of the bits of the local data bus, a second data input coupledto the another half of the bits of the local data bus, a multiplexedoutput coupled to the parallel input of the first PISO shift register,and a select input coupled to a data bus select signal, the busmultiplexer to selectively couple the one half of the bits of the localdata bus or the another half of the bits of the local data bus into theparallel input of the first PISO shift register in response to the databus select signal.
 8. The integrated circuit of claim 7, wherein each ofthe one or more lanes further includes control logic coupled to themultiplexer and the first PISO shift register, the control logic toreceive the first clock signal, a mode select signal, and a merge enablesignal, the control logic to generate the local data select signal tomerge the serialized local data and the feed-through data into theserial data stream in response to the merge enable signal and the firstclock signal, the control logic to further generate the data bus selectsignal in response to the mode select signal.
 9. The integrated circuitof claim 8, wherein: the load signal is an early load pulse signal thatis coupled into the load/shift-bar bar input of the second PISO shiftregister, and the control logic has merge control logic that includes: asecond multiplexer having a first data input coupled to the merge enablesignal and a select input coupled to the early load pulse signal: afirst D-type flip-flop coupled to the second multiplexer, the firstD-type flip-flop having a data input coupled to an output of the secondmultiplexer, and a clock input coupled to the first clock signal, and adata output coupled to the select input of the first multiplexer and asecond data input of the second multiplexer, the first D-type flip-flopto register the merge enable signal in response to the early load pulsesignal and the first clock signal to generate the local data selectsignal on the data output: and wherein the second multiplexer torecirculate the local data select signal into the data input of thefirst D-type flip-flop in response to the early load pulse signal beinga logical low and to couple the merge enable signal into the firstD-type flip-flop in response to the early load pulse signal being alogical high; the control logic further has mode control logic thatincludes: a third multiplexer having a first data input coupled to theearly load pulse signal: a second D-type flip-flop coupled to the thirdmultiplexer, the second D-type flip-flop having a data input coupled toan output of the third multiplexer, a clock input coupled to the firstclock signal, a clear input coupled to an inverted bus mode signal, anda data output coupled to the select input of the bus multiplexer and asecond data input of the third multiplexer, the second D-type flip-flopto generate the data bus select signal on the data output in response tothe inverted bus mode signal, the early load pulse signal, and the firstclock signal: an OR gate having a first input coupled to the early loadpulse signal and a second input coupled to a late load pulse signal, theOR gate to logically OR the early load pulse signal and the late loadpulse signal: an AND gate having a first input coupled to an output ofthe OR gate, a second input coupled to a bus mode signal, an outputcoupled to a select input of the third multiplexer: an inverter havingan input coupled to the bus mode signal and an output coupled to theclear input of the second D-type flip-flop, the inverter to generate theinverted bus mode signal in response to the bus mode signal: a fourthmultiplexer having a first data input coupled to the early load pulsesignal, a second data input coupled to the output of the OR gate, acontrol input coupled to the bus mode signal, and a multiplexed outputcoupled to the load/shift-bar bar input of the first PISO shiftregister, wherein the third multiplexer to recirculate the data busselect signal into the data input of the second D-type flip-flop inresponse to the inverted bus mode signal: and wherein the fourthmultiplexer to selectively couple the early load pulse signal or boththe early load pulse signal and the late load pulse signal into theload/shift-bar bar input of the first PISO shift register.
 10. Theintegrated circuit of claim 9, wherein: the integrated circuit is abuffer integrated circuit: the local data bus is twelve bits wide: andthe bus multiplexer in each of the one or more lanes selectively couplesa lower six bits of the local bus into the first PISO shift register andan upper six bits of the local bus into the first PISO shift register inresponse to the data bus select signal.
 11. A method for a memory modulecomprising: receiving an input serial data stream representingfeed-through frames of data interspersed between idle frames of data;merging local frames of data and the feed-through frames of datatogether into an output serial data stream in response to a merge enablesignal without decoding the input serial data stream; and transmittingthe output serial data stream on a northbound data output to a nextmemory module or a memory controller.
 12. (canceled)
 13. The method ofclaim 11, wherein the receiving of the input serial data stream includessampling the bits of data in the input serial data stream.
 14. Themethod of claim 11, wherein the receiving of the input serial datastream includes re-synchronizing the bits of data in the input serialdata stream.
 15. The method of claim 11, wherein the merging of thelocal frames of data and the feed-through frames of data togetherincludes: serializing parallel bits of the local frames of data intoserial bits of data: and multiplexing the serial bits of data of thelocal frames of data and serial bits of the feed-through frames of datainto serial bits of the output serial data stream in response to themerge enable signal.
 16. The method of claim 11, wherein a local frameof data of the local frames of data is selectively received in parallelover a local bus in six bit or twelve bit packets in response to a busmode signal.
 17. A system comprising: a processor to executeinstructions and process data; a memory controller coupled to theprocessor, the memory controller to receive write memory instructionswith write data from the processor and to receive read memoryinstructions from the processor and supply read data to the processor;at least one bank of memory coupled to the memory controller, the atleast one bank of memory including one or more memory modules, each ofthe one or more memory modules including: a buffer integrated circuit:and at least one memory integrated circuit; and wherein the bufferintegrated circuit includes: a southbound serial input/output interfacewith one or more serial lanes to receive the write data from the memorycontroller: and a northbound serial input/output interface with one ormore serial lanes of a northbound serial input and a northbound serialoutput, the northbound serial input/output interface to transmit theread data to the memory controller, the northbound serial input/outputinterface including for each of the one or more serial lanes including:a parallel-to-serial converter having a parallel input coupled toparallel bits of a local data bus, a clock input coupled to a firstclock signal, a load/shift-bar input coupled to a load signal, theparallel-to-serial converter to serialize the parallel bits of data onthe local data bus into serialized local data on a first serial output;and a first multiplexer having a first data input coupled to the firstserial output of the parallel-to-serial converter, a second data inputto receive serial feed-through data from the northbound serial input,and a select input coupled to a local data select signal, themultiplexer to selectively merge the serialized local data and theserial feed-through data into a serial data stream on the northboundserial output in response to the local data select signal.
 18. Thesystem of claim 17, wherein each serial lane of the northbound serialinput/output interface further includes a transmitter having an inputcoupled to the multiplexed output of the first multiplexer to receivethe serial data stream, the transmitter to drive the serial data streamonto the northbound serial data output towards the memory controller.19. The system of claim 17, wherein: for each bank of memory, the memorycontroller includes a northbound serial input interface to receive oneor more lanes of serial data from the one or more memory modules: and asouthbound serial output interface to transmit one or more lanes ofserial data to the one or more memory modules.
 20. The system of claim17, wherein each of the one or more serial lanes of the northboundserial input/output interface further includes control logic coupled tothe multiplexer and the parallel-to-serial converter, the control logicto receive the first clock signal and a merge enable signal, the controllogic to generate the local data select signal to merge the serializedlocal data and the serial feed-through data into the serial data streamin response to the merge enable signal and the first clock signal. 21.(canceled)
 22. A buffered memory module comprising: a printed circuitboard with an edge connection; a plurality of memory integrated circuitscoupled to the printed circuit board; and a buffer integrated circuitcoupled to the printed circuit board, the buffer integrated circuitfurther electrically coupled to the plurality of memory integratedcircuits and the edge connection, the buffer integrated circuit having asouthbound input/output interface and a northbound input/outputinterface, the northbound input/output interface includes data mergelogic with a plurality of merge logic slices for a plurality of lanes ofserial data streams, each of the plurality of merge logic slicesincluding: a first parallel-in-serial-output (PISO) shift registerhaving a parallel input coupled to a local data bus, a clock inputcoupled to a first clock signal, a load/shift-bar input coupled to afirst load signal, the first PISO shift register to serialize paralleldata on the local data bus into serialized local data on a first serialoutput; and a first multiplexer having a first data input coupled to thefirst serial output of the first PISO shift register, a second datainput to receive serialized feed-through data, a select input coupled toa local data select signal, the first multiplexer to selectively mergethe serialized local data and the serialized feed-through data into aserial data stream on a multiplexed output in response to the local dataselect signal.
 23. The buffered memory module of claim 22, wherein thenorthbound input/output interface further includes a plurality oftransmitters each having an input coupled to a corresponding output ofthe first multiplexer in each of the plurality of merge logic slices,the plurality of transmitters to receive the serial data stream anddrive it onto a serial data link.
 24. (canceled)
 25. The buffered memorymodule of claim 22, wherein each of the plurality of merge logic slicesfurther includes control logic coupled to the first multiplexer and thefirst PISO shift register, the control logic to receive the first clocksignal and a merge enable signal, the control logic to generate thelocal data select signal to merge the serialized local data and theserialized feed-through data into the serial data stream in response tothe merge enable signal and the first clock signal.
 26. A memory systemcomprising: a plurality of buffered memory modules daisy chainedtogether to form a bank of memory, each buffered memory moduleincluding: a plurality of memory integrated circuits; and a bufferintegrated circuit coupled to the plurality of memory integratedcircuits, the buffer integrated circuit including: a southboundinput/output serial interface to receive and retransmit southboundserial data from a memory controller or a prior buffered memory moduleto a next buffered memory module: a northbound input/output serialinterface to receive northbound serial data from at least one bufferedmemory module as serialized feed-through data and retransmit it towardsthe memory controller, the northbound input/output serial interface toserialize local data from the plurality of memory integrated circuitsand merge it into a northbound serial data stream with the serializedfeed-through data without decoding the received northbound serial data,the northbound input/output serial interface further to transmit thenorthbound serial data stream, including the serialized feed-throughdata and the serialized local data, towards the memory controller: awrite data first-in-first-out (FIFO) buffer coupled to the southboundinput/output serial interface, the write data FIFO buffer to store writedata from the southbound input/output serial interface addressed to thebuffered memory module by a write command: a memory input/outputinterface coupled to the plurality of memory integrated circuits and thewrite data FIFO buffer, the memory input/output interface to transferwrite data stored in the write data FIFO buffer into at least one of theplurality of memory integrated circuits, a read data FIFO buffer coupledto the northbound input/output serial interface and the memoryinput/output interface, the read data FIFO buffer to store read datafrom at least one of the plurality of memory integrated circuits as thelocal data addressed from the buffered memory module by a read command:and wherein the memory input/output interface to transfer read data fromthe at least one of the plurality of memory integrated circuits into theread data FIFO buffer.
 27. The memory system of claim 26, furthercomprising the memory controller coupled to at least one of theplurality of buffered memory modules, the memory controller including: asouthbound output serial interface to transmit the southbound serialdata stream to the at least one of the plurality of buffered memorymodules: and a northbound input serial interface to receive thenorthbound serial data stream from the at least one of the plurality ofbuffered memory modules.
 28. The memory system of claim 26, wherein thenorthbound input/output serial interface of the buffer integratedcircuit includes: a third FIFO buffer: data merge logic coupled to thethird FIFO buffer, the data merge logic having a plurality of mergelogic slices each including: a first parallel-in-serial-output (PISO)shift register having a parallel input coupled to a local data bus, aclock input coupled to a first clock signal, a load/shift-bar inputcoupled to a first load signal, the first PISO shift register toserialize parallel data on the local data bus into serialized local dataon a first serial output: and a first multiplexer having a first datainput coupled to the first serial output of the first PISO shiftregister, a second data input to receive serialized feed-through data, aselect input coupled to a local data select signal, the firstmultiplexer to selectively merge the serialized local data and theserialized feed-through data into a serial data stream on a multiplexedoutput in response to the local data select signal, and a plurality oftransmitters coupled to the data merge logic, each of the plurality oftransmitters having an input coupled to a corresponding output of thefirst multiplexer in each merge logic slice, the plurality oftransmitters to receive the serial data stream and drive it onto aserial data link.
 29. The memory system of claim 28, wherein each mergelogic slice of the data merge logic further includes a secondmultiplexer having a first data input to receive resynchronized serialdata, a second data input to receive re-sampled serial data, a selectinput coupled to a local clock mode signal, the second multiplexer toselect the re-sampled serial data or the resynchronized serial data asthe serialized feed-through data in response to the local clock modesignal.
 30. The memory system of claim 28, wherein each merge logicslice of the data merge logic further includes control logic coupled tothe first multiplexer and the first PISO shift register, the controllogic to receive the first clock signal and a merge enable signal, thecontrol logic to generate the local data select signal to merge theserialized local data and the serialized feed-through data into theserial data stream in response to the merge enable signal and the firstclock signal.